Obesity is closely related to non-alcoholic fatty liver disease (NAFLD). Although sex differences in body fat distribution have been well demonstrated, little is known about the sex-specific associations between adipose tissue and the development of NAFLD. Using community-based cohort data, we evaluated the associations between magnetic resonance imaging quantified areas of abdominal adipose tissue, including visceral adipose tissue (VAT) and subcutaneous adipose tissue (SAT), and incident NAFLD in 2830 participants (1205 males and 1625 females) aged 55–70 years. During a 4.6-year median follow-up, the cumulative incidence rates of NAFLD increased with areas of VAT and SAT both in males and in females. Further analyses showed that the above-mentioned positive associations were stronger in males than in females, especially in participants under 60 years old. In contrast, these sex differences disappeared in those over 60 years old. Furthermore, the risk of developing NAFLD increased non-linearly with increasing fat area in a sex-specific pattern. Additionally, sex-specific potential mediators, such as insulin resistance, lipid metabolism, inflammation, and adipokines, may exist in the associations between adipose tissue and NAFLD. This study showed that the associations between abdominal fat and the risk of NAFLD were stratified by sex and age, highlighting the potential need for sex- and age-specific management of NAFLD.